Description

Previously when trying to run with 'rdma_mpi' equal to true, I would get illegal buffer pointer values in the halo communication routines. After a small change that makes sense to me, CUDA-aware MPI now seems to work on both Delta and Phoenix.

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Something else

Scope

• This PR comprises a set of related changes with a common goal

If you cannot check the above box, please split your PR into multiple PRs that each have a common goal.

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.
Provide instructions so we can reproduce.
Please also list any relevant details for your test configuration

I built and ran 2 cases that differed only by the 'rdma_mpi' status on both Phoenix and Delta. On Delta, it was 3D_TGV and on Phoenix it was 2D_riemman test. h5diff on the resulting hd5f files of the last step showed no difference on Phoenix. Additionally, I took nsys reports on phoenix to show the lack of H2D/D2H copies.

First without RDMA_MPI, the code moves the GPU data to the CPU before MPI exchange:

With RDMA_MPI enabled, there are no data copies. This led to a marginal improvement in the RHS-MPI time for this case:

If your code changes any code source files (anything in src/simulation)

To make sure the code is performing as expected on GPU devices, I have:

• Checked that the code compiles using NVHPC compilers
• Checked that the code compiles using CRAY compilers
• Ran the code on either V100, A100, or H100 GPUs and ensured the new feature performed as expected (the GPU results match the CPU results)
• Ran the code on MI200+ GPUs and ensure the new features performed as expected (the GPU results match the CPU results)
• Enclosed the new feature via nvtx ranges so that they can be identified in profiles
• Ran a Nsight Systems profile using ./mfc.sh run XXXX --gpu -t simulation --nsys, and have attached the output file (.nsys-rep) and plain text results to this PR
• Ran an Omniperf profile using ./mfc.sh run XXXX --gpu -t simulation --omniperf, and have attached the output file and plain text results to this PR.
• Ran my code using various numbers of different GPUs (1, 2, and 8, for example) in parallel and made sure that the results scale similarly to what happens if you run without the new code/feature